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Abstract. We study tree languages that can be defined in A2. These are tree languages 
definable by a first-order formula whose quantifier prefix is 3*V*, and simultaneously by a 
first-order formula whose quantifier prefix is V*3*. For the quantifier free part we consider 
two signatures, either the descendant relation alone or together with the lexicographical 
order relation on nodes. We provide an effective characterization of tree and forest lan- 
guages definable in A2. This characterization is in terms of algebraic equations. Over 
words, the class of word languages definable in A2 forms a robust class, which was given 
an effective algebraic characterization by Pin and Weil 



1. Introduction 

We say a logic £1 has a decidable characterization inside a logic £2 if the following 
decision problem is decidable: "given as input a formula of the logic £2, decide if it is 
equivalent to some formula of the logic £1". We are interested in the case when the logic 
£2 is MSO on words or trees, and £1 represents some fragment of £2- 

This type of problem has been successfully studied in the case when £2 is MSO on finite 
words. In other words £2, represents the class of regular word languages. Arguably best 
known is the result of McNaughton, Papert and Schiitzenberger |13| 19], which says that the 
following two conditions on a regular word language L are equivalent: a) L can be defined in 
first-order logic with order and label tests; b) the syntactic semigroup of L does not contain 
a non-trivial group. Since condition b) can be effectively tested, the above theorem gives a 
decidable characterization of first-order logic. This result demonstrates the importance of 
this type of work: a decidable characterization not only gives a better understanding of the 
logic in question, but it often reveals unexpected connections with algebraic concepts. Dur- 
ing several decades of research, decidable characterizations have been found for fragments 
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of first-order logic with restricted quantification and various signatures (typically subsets of 
the order relation and the successor relation), as well as a large group of temporal logics, 
see |10j and [T7] for references. 

An important part of this research has been devoted to the quantifier alternation hier- 
archy, where each level counts the alterations between V and 3 quantifiers in a first-order 
formula in prenex normal form. The quantifier free part of such a formula is built using 
a binary predicate < representing the linear order on the word. Formulas that have n — 1 
alternations (and therefore n blocks of quantifiers) are called S n (<) if they begin with 3, 
and II n (<) if they begin with V. For instance, the word property "some position has label 
a" can be defined by a £i(<) formula 3x. a(x), while the language "nonempty words with 
at most two positions that do not have label o" can be defined by the £2(<) formula 

3xi3x 2 Vy (y ^ x 1 A y / x 2 ) => a(y) . 

A lot of attention has been devoted to analyzing the low levels of the quantifier alter- 
nation hierarchy for word languages. The two lowest levels are easy: a word language is 
definable in £i(<) (resp. Ill (<)) if and only if it is closed under inserting (removing) let- 
ters. Both properties can be tested in polynomial time based on a recognizing automaton, or 
semigroup. However, just above Si(<), IIi (<), and even before we get to S2(<), Il2(<), we 
already find two important classes of languages. A fundamental result, due to Simon |15j . 
says that a language is defined by a Boolean combination of £i(<) formulas if and only if 
its syntactic monoid is ^-trivial. Above the Boolean combination of £i(<), we find A2(<), 
i.e. languages that can be defined simultaneously in £2(<) and Il2(<). As we will describe 
later on, this class turns out to be surprisingly robust, and it is the focus of this paper. 
Another fundamental result, due to Pin and Weil [11], says that a regular language is in 
A2(<) if and only if its syntactic monoid is in DA. The limit of our knowledge is level 
S2(<): it is decidable if a language can be defined on level ^2(<) [HE], but there are no 
known decidable characterization for Boolean combinations of Yi2(<), for As(<), for S3(<), 
and upwards. 

For trees even less is known. No decidable characterization has been found for what 
is arguably the most important proper subclass of regular tree languages, first-order logic 
with the descendant relation, despite several attempts. Similarly open are chain logic and 
the temporal logics CTL, CTL* and PDL. However, there has been some recent progress. 
In [5J, decidable characterizations were presented for some temporal logics, while Benedikt 
and Segoufin [2] characterized tree languages definable in first-order logic with the successor 
relation (but without the descendant relation). 

This paper is part of a program to understand the expressive power of first-order logic 
on trees, and the quantifier alternation hierarchy in particular. The idea is to try to under- 
stand the low levels of the quantifier alternation hierarchy before taking on full first-order 
logic (which is contrary to the order in which word languages were analyzed). We focus 
on two signatures. The first signature contains unary predicates for label tests and the 
ancestor order on nodes, denoted <. The second signature assumes that the trees have an 
order on siblings, which induces a lexicographical linear order on nodes, denoted <i ex - Both 
signatures generalize the linear order on words. As shown in [3], there is a reasonable notion 
of concatenation hierarchy for tree languages that corresponds to the quantifier alternation 
hierarchy. Levels £i(<) and ni(<) are as simple for trees as they are for words. A recent 
result [7] extends Simon's theorem to trees, by giving a decidable characterization of tree 
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languages definable by a Boolean combination of Si(<) formulas, and also a decidable char- 
acterization of Boolean combinations of Si(<, <i cx ) formulas. There is no known decidable 
characterization of tree languages definable in S n (<) for n > 2. 

The contribution of this paper is a decidable characterization of tree languages de- 
finable in A2(<), i.e. definable both in X2(<) and LT2(<). We also provide a decidable 
characterization of tree languages definable in A2(<, <i ex )- 

As we signaled above, for word languages the class A2(<) is well studied and important, 
with numerous equivalent characterizations. Among them one can find \16 \ [T^ l 18]: a) 
word languages that can be defined in the temporal logic with operators F and F~ , b) 
word languages that can be defined by a first-order formula with two variables, but with 
unlimited quantifier alternations; c) word languages whose syntactic semigroup belongs to 
the semigroup variety DA; d) word languages recognized by two-way ordered deterministic 
automata; e) a certain form of "unambiguous" regular expressions. 

It is not clear how to extend some of these concepts to trees. Even when natural tree 
counterparts exist, they are not equivalent. For instance, the temporal logic in a) can 
be defined for trees — by using operators "in some descendant" and "in some ancestor". 
This temporal logic was studied in [3], however it was shown to have an expressive power 
incomparable with that of A2(<). A characterization of A2(<) was left as an open problem, 
one which is solved here. 

We provide an algebraic characterization of tree languages definable in A2(<) and in 
A2(<, <iex)- This characterization is effectively verifiable if the language is given by a tree 
automaton. It is easy to see that the word setting can be treated as a special case of the 
tree setting. Hence our characterization builds on the one over words. However the added 
complexity of the tree setting makes both formulating the correct condition and generalizing 
the proof quite nontrivial. 

2. Trees forests and languages 

In this section, we present some basic definitions regarding trees. We also present the 
formalism of forest algebra, which is used in our characterizations. 

2.1. Trees, forests and contexts. In this paper we work with finite, unranked, ordered 
trees and forests over a finite alphabet A. Formally, these are expressions defined inductively 
as follows: If s is a forest and a £ A, then as is a tree. If t\, . . . , t n is a finite sequence of 

trees, then ti-\ \-t n is a forest. This applies as well to the empty sequence of trees, which 

is called the empty forest, and denoted (and which provides a place for the induction to 
start). Forests and trees alike will be denoted by the letters s,t,u, . . . When necessary, we 
will remark which forests are trees, i.e. contain only one tree in the sequence. 

The notion of node, as well as the descendant and ancestor relations are defined in the 
usual way. We write x < y to say that x is a strict ancestor of y or, equivalently, that y is 
a strict descendant of x. As usual, we write x < y when x = y or x < y. The parent of 
a node x is its immediate ancestor. Two nodes x and y are siblings if they have the same 
parent. We also use the lexicographic order on nodes, written <i ex - Recall that x <\ ex y 
holds if either x < y, or there are nodes x' < x and y' < y such that x' is a sibling to the 
left of y'. 

If we take a forest and replace one of the leaves by a special symbol □, we obtain a 
context. Contexts will be denoted using letters p, q, r. A forest s can be substituted in 
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place of the hole of a context p, the resulting forest is denoted by ps. There is a natural 
composition operation on contexts: the context qp is formed by replacing the hole of q with 
p. This operation is associative, and satisfies (pq)s = p(qs) for all forests s and contexts p 
and q. 

We say a forest s is an immediate piece of a forest s' if s, s' can be decomposed as s = pt 
and s' = pat for some context p, some label a, and some forest t. The reflexive transitive 
closure of the immediate piece relation is called the piece relation. We write s ^ t to say 
that s is a piece of t. In other words, a piece of t is obtained by removing nodes from t. 
We extend the notion of piece to contexts. In this case, the hole must be preserved while 
removing the nodes. The notions of piece for forests and contexts are related, of course. 
For instance, if p, q are contexts with p X q, then pO H qO. Also, conversely, if s X t, then 
there are contexts p ^ q with s = pO and t = qO. (For instance, one can take p = □ + s and 
q = □ + 1.) The picture below depicts two contexts, the left one being a piece of the right 
one, as can be seen by removing the white nodes. 




We will be considering three types of languages in the paper: forest languages i.e. sets of 
forests, denoted L; context languages, i.e. sets of contexts, denoted K, and tree languages, 
i.e. sets of trees, denoted M. Note that a forest language can contain trees. 

2.2. Forest algebras. Forest algebras were introduced by Bojahczyk and Walukiewicz as 
an algebraic formalism for studying regular tree languages [6] . Here we give a brief summary 
of the definition of these algebras and their important properties. A forest algebra consists of 
a pair (H, V) of finite monoids, subject to some additional requirements, which we describe 
below. We write the operation in V multiplicatively and the operation in H additively, 
although H is not assumed to be commutative. We accordingly denote the identity of V 
by □ and that of H by 0. We require that V act on the left of H. That is, there is a map 
(h,v) i->- vh G H such that w(vh) = {wv)h for all h G H and v,w 6 V. We further require 
that this action be monoidal, that is, □ • h = h for all h £ H, and that it be faithful, that is, 
if vh = wh for all h 6 H, then v = w. Finally we require that for every g 6 H, V contains 
elements (□ + g) and (g + □) defined by (□ + g)h = h + g,{g + 0)h = g + h for all h £ H. 

A morphism a : (Hi, V\) — > (H2, Vjj) of forest algebras is actually a pair (an, ay) where 
an is a monoid morphism between Hi and H2 and ay is a monoid morphism between V\ 
and V2, such that an(vh) = av{v)aji(h) for all /i € H, v € V. However, we will abuse 
notation slightly and denote both component maps by a. 

Let A be a finite alphabet, and let us denote by Ha the set of forests over A, and 
by Va the set of contexts over A. Each of these is a monoid, with the operations being 
forest concatenation and context composition, respectively. The pair (Ha,Va), with forest 
substitution as action, forms a forest algebra, which we denote A A . 

We say that a forest algebra (H, V) recognizes a forest language L C Ha if there is a 
morphism a : A A — > (H, V) and a subset X of H such that L = a^ 1 (X). A forest language 
is regular, i.e. recognized by any of the many equivalent notions of automata for unranked 
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trees that can be found in the literature, if and only if it is recognized by a finite forest 
algebra [6]. 

Given any finite monoid M, there is a number uj(M) (denoted by oj when M is under- 
stood from the context) such that for all elements x of M, x u is an idempotent: x u = x^x^ . 
Therefore for any forest algebra (H, V) and any element u of V and g of H we will write 
u w and ujg for the corresponding idempotents. The element is idempotent with respect 
to the operation in V . The element cog, which is the same as (□ + g) u and the same as 
{g + □) w 0, is idempotent with respect to the operation in H. 

Given a forest language L C Ha we define an equivalence relation ~t on Ha by setting 
s ~l s' if and only if for every context p E Va, the forests ps and ps' are either both in 
L or both outside of L. We further define an equivalence relation on Va, also denoted ~l, 
by setting p ~^ p' if for all s G Ha, ps ~l p's. This pair of equivalence relations defines a 
congruence of forest algebras on A A , and the quotient (Hl, V£) is called the syntactic forest 
algebra of L. Each equivalence class of ~l is called a type. 

We now extend the notion of piece to elements of a forest algebra (H, V). The general 
idea is that a context type v G V is a piece of a context type w G V if one can construct 
a term (using elements of -ff and V) which evaluates to w, and then take out some parts 
of this term to get v. Let (H, V) be a forest algebra. We say v G V is a piece of w G V, 
denoted by v X u>, if there is an alphabet ^4 such that a(p) = v and a(g) = w hold for some 
morphism 

a : A A (JT, V) 

and some contexts p ^ q over A. The relation ^ is extended to by setting g ^ /i if g = vO 
and /i = wO for some context types v <w. 

3. Logic 

The focus of this paper is the expressive power of first-order logic on trees. A forest can 
be seen as a logical relational structure. The domain of the structure is the set of nodes. (We 
allow empty domains, which happens when an empty forest is considered.) We consider 
two different signatures. Both of them contain a unary predicate P a for each symbol a of 
the alphabet A, as well as a binary predicate < for the ancestor relation. Furthermore, 
the second signature also contains a binary predicate <i ex for the lexicographic order on 
nodes. A formula without free variables over these signatures defines a set of forests, these 
are the forests where it is true. We are particularly interested in formulas of low quantifier 
complexity. A T,2 formula is a formula of the form 

3xi • • • x n Vyi • • • y m 7 , 

where 7 is quantifier free. Languages defined in £2 are closed under disjunction and con- 
junction, but not necessarily negation. The negation of a £2 formula is called a II2 formula, 
equivalently this is a formula whose quantifier prefix is V*3*. A forest property is called A2 
if it can be expressed both by a £2 and a LT2 formula. We will use £a(<) and £2(<, <iex) 
to specify which predicates are used in the signature, similarly for II2 and A2. 

With limited quantification, the choice of signature is a delicate question. For instance, 
adding a child relation changes the expressive power. 
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3.1. The problem. We want an algorithm deciding whether a given regular forest language 
is definable in A2(<, <i ex ) an d another one for deciding whether it is in A2(<). 

As noted earlier, the corresponding problem for words was solved by Pin and Weil |llj : 
a word language L is definable in A2(<) if and only if its syntactic monoid M(L) belongs 
to the variety DA, i.e. it satisfies the identity 

(mn) u = [mn)^ 'm(mn) u 

for all m, n S M(L). The power u means that the identity holds for sufficiently large powers 
(in different settings, u) is defined in terms of idempotent powers, but the condition on 
sufficiently large powers is good enough here) . Since one can effectively test if a finite monoid 
satisfies the above property (it is sufficient to verify the power \M(L)\), it is decidable 
whether a given regular word language is definable in A2(<). We assume that the language 
L is given by its syntactic monoid and syntactic morphism, or by some other representation, 
such as a finite automaton, from which these can be effectively computed. 

We will show that a similar characterization can be found for forests; although the 
identities will be more involved. For decidability, it is not important how the input language 
is represented. In this paper, we will represent a forest language by a forest algebra that 
recognizes it. Forest algebras are described in the next section. 

3.2. Tree languages. We give an algorithm which says when a forest language belongs to 
a class C, which is either A2(<) or A2(<,<i ex )- What about tree languages? There are 
two ways of getting a class of tree languages from a class of forest languages C. 

(1) The class of tree languages that belong to C 

(2) The class of tree languages of the form L n Ta, where A is an alphabet, Ta is the set of 
all trees over alphabet A, and L £ C is a forest language over A. 

Our algorithm gives a decision procedure under the first definition. The usual understanding 
of tree languages definable in A2(<) or A2(<, <i cx ) corresponds to the second definition. 

Fortunately, the two definitions are equivalent when C is either A2(<) or A2(<, <iex)- 
This is because in both cases, C is closed under intersection and contains the languages Ta- 

Closure under intersection is immediate. Why does C contain the languages Tj$ Since 
A2(<) is the less powerful logic, it suffices to show how to define Ta using a ^2(<) formula, 
and also using a Il2(<) formula. The £2(<) formula says there exists a node that is an 
ancestor of all other nodes, while the Il2(<) formula says that for every two nodes, there 
exists a common ancestor. 

In general, the definitions of tree language classes are not equivalent. Consider as C the 
class of forest languages defined by purely existential formulas. In particular, if a language 
L G C contains a tree t, then it also contains the forest t + t. This means that under the first 
definition, the only tree languages we get are the empty tree languages. Under the second 
definition, we get some more tree languages, such as "trees with at least two nodes". 

3.3. Basic properties of IIi and £2. Most of the proofs in the paper will work with 
^2(<) ° r ^2(< ; <iex) formulas. We present some simple properties of such formulas in this 
section. 

Apart from defining forest languages, we will also be using formulas to define languages 
of contexts. To define a context language we use formulas with a free variable; such a 
formula is said to hold in a context if it is true when the free variable is mapped to the hole 



TREE LANGUAGES DEFINED IN FIRST-ORDER LOGIC WITH ONE QUANTIFIER ALTERNATION 7 



of the context. § For instance, the formula Vy y < x => a(y) with a free variable x defines 
the set of contexts where every ancestor of the hole has label a. 

We begin by describing the expressive power of purely universal formulas III. 

Lemma 3.1. A forest language is closed under pieces if and only if it is definable in U\. 
Likewise for context languages. 

Proof. It is clear that a forest language definable in 111 is closed under pieces as the models 
of a ITi formula are closed under substructures. 

For the converse, let L be a forest language that is closed under pieces, and let a : 
—7- (H, V) be its syntactic algebra. Thanks to a pumping argument, any forest has a 
piece with the same type, but at most nodes. Let T be the finite set of forests with at 

most nodes that are outside L. Thanks to the pumping argument, a forest belongs to 

L if and only if it has no piece in the set T. The latter is a property that can be expressed 
in IIi(<, <i cx ). 

If, additionally, the language L is commutative, then we do not need to worry about 
the lexicographic order when talking about the pieces in T, only the descendant order is 
relevant. D 

We now turn to the formulas from £2. We begin with £2(<> <iex)j since it has the 
better closure properties. 

Lemma 3.2. Let K, K' be context languages and L, L' be forest languages. Lf these lan- 
guages are all definable in 5j2(<, <lex); then so are 

(1) the forest language KL = {qt : q G K,t G L], 

(2) the forest language L + L' = {t + 1' : t G L,t' G L'}, 

(3) the context language KK' = {qq' : q G K,q' G K'}, 

(4) the forest language L n L' = {t : t G L,t G L'}, 

(5) the context language K n K' = {q : q G K, q G K'}, 

Proof. We only do the proof for KL, the others are treated similarly. When does a forest t 
belong to KL? There must exist two siblings x\ and X2 such that the set, call it X, of gray 
nodes in the picture below 



describes a forest in L, and the other nodes describe a context in K. Below we define this 
property more precisely, and show that it can be defined in T>2(<, <i cx )- 

First, we want to say that the nodes x\ and X2 are siblings. This can be expressed by 
a formula, call it a(xi,X2), of 5j2(<) <icx)- The formula quantifies existentially a common 
ancestor and uses universal quantification to check that this common ancestor is a parent 
of both nodes: 

a(xi, X2) = 3x Vy x% <i ex X2 A x < x\ A x < X2 A {y < x\ — > y < x) A (y < X2 — > y < £2) 




L 
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Next, we describe the set X. Membership x £ X is defined by a quantifier- free formula 
fi(x) = (xi <i ex x A x <i ex x 2 ) V (x > x 2 ). 

Next, we say what it means for the part inside X describes a forest in L. Suppose that 
the forest language L is defined by a formula <p of T, 2 (<, <i ex )- To say that X describes a 
forest in L we use the formula <p" obtained from tp by restricting quantification to nodes 
satisfying (3. Note that ip 13 has free variables x\,x 2 from /3. Since /3 is quantifier- free, the 
formula <fP is also in £2(<, <i cx )- 

Finally, we say that the part outside X describes a context in K. The idea is that the 
hole of this context corresponds to the set X. The logical formula is constructed below. 
Suppose that <j>(x) is a formula of T, 2 (<, <i ox ) that describes K. Note that this formula has 
a free variable, as with formulas for contexts, which corresponds to the hole. Let (^"(x) be 
the formula obtained from 4>{x) by restricting quantification to nodes not satisfying /3. The 
remaining question is: which node should we use for x? Any node from X will do, since 
for each node y X, all nodes from X have the same relationship to y, with respect to the 
descendant and lexicographic orders. We use the node x\ for x. 

Summing up, the formula for KL is written below. 

3xi3x 2 a{xi,x 2 ) A 4>^{xi) A</A □ 



3.4. £2 expressions. In the proofs, it will sometimes be more convenient to use a type 
of regular expression instead of formulas. These are called £2 forest expressions and £2 
context expressions, and are defined below by mutual recursion: 

• Any forest (respectively, context) language that is closed under pieces is a £2 forest 
(respectively, context) expression. For any label a £ A, {ad} is a £2 context expression. 
Likewise for {□}, the language containing only the empty context. 

• If K, K' are £2 context expressions and L, V are £2 forest expressions, then 

— K ■ K' is a Ti 2 context expression; 

— L + L' is a £2 forest expression; 

— K ■ L is a £2 forest expression. 

— L U L' is a £2 forest expression. 

— K U K' is a £2 forest expression. 

— L n L' is a £2 forest expression. 

— K n K' is a £2 forest expression. 



From Lemmas 3.1 and |3.2| it follows that languages defined by £2 forest and context ex- 



pressions are definable in £2(<, <i cx )- 

4. Characterization of A 2 (<,<i ex ) 
The main result of this paper is the following theorem: 
Theorem 4.1 (Effective characterization of A2 with descendant and lexicographic orders). 

A forest language is definable in A 2 (<, <i ex ) if and only if its syntactic forest algebra satisfies 
the following identity, called the A 2 identity, 

v^wv" = v" forw^v . (4.1) 
Before we prove the main theorem, we state and prove an important corollary. 
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Corollary 4.2. It is decidable whether a forest language can be defined in A2(<,<i ex )- 

Proof. We assume that the language is represented as a forest algebra. This representation 
can be computed based on other representations, such as automata or monadic second-order 
logic. 

Once the forest algebra is given, the A2 identity can be tested in polynomial time by 
searching through all elements of the algebra. The relation ^ can be computed in polynomial 
time, using a fixpoint algorithm as in [3]. O 

The following lemma gives the easier implication from the main theorem. 



Lemma 4.3. Let if be a formula of S2(<, <i cx ) an d let q ^ p be two contexts. For n 6 N 
sufficiently large, forests satisfying p are closed under replacing p n p" with p n qp n 



Proof. Assume that p is 3xi ■ ■ ■ XkVyi ■ ■ ■ yiip, with ip quantifier- free. Any first-order defin- 
able tree language is aperiodic |13j . i.e. there is a number m such that any context p % can 
be replaced by p 1 without affecting membership in the language, for any i, j > m. We set 
n = 2m + (k + l)(l + 1). 

Consider a forest t = rp n p n s that satisfies (p. We want to show that the forest rp n qp n s 
also satisfies ip. By aperiodicity, it is sufficient to show that for some numbers i,j > m, 
rp l qpP s satisfies (p. 

Because t satisfies ip W6 can fix h nodes xi, . . . , x& that make Vyi . . . yiip true. By the 
choice of n, t can be decomposed as rp % p l pP s such that i,j > m and the middle p l part 
contains none of the nodes x%, . . . , x^. We show that t' = rp % p l qp > s satisfies <p, which will 
conclude the proof of the lemma. 

We identify the nodes of t with the nodes of t' outside the inserted context q. Consider 
the valuation of the variables x\, ■ ■ ■ ,Xk that we fixed above. We show that this valuation, 
when seen as nodes of t! , makes p> true. Indeed, for any valuation for the variables yi, • • • , y\ 
in t' , we show that the valuation makes the quantifier- free part tp true. This is obvious if 
none of the yi are in q because p holds in t and the insertion of q does not affect the 
relationship < and <i ex between the selected nodes. If some of the yi are in q then one 
of the contexts p in the middle block p l of t! does not contain any variable. As removing 
the context p that does not contain any variable does not affect the relationship < and 
<iex between the selected nodes, ip(xi,--- - ,yi) holds on rp l p l qp>s iff it holds 

on rp l p l ~ 1 qpi s. Moreover, because q is a piece of p, replacing q by p does not affect the 
relationship < and <i ex between the selected nodes and therefore ip(x\, • • • , Xk, yi," - ,Ui) 
holds on rp % p l ~^qpP s iff it holds on rp l p l ~ l ppi s = t. Hence ip must hold with the new 
valuation. O 



The rest of the paper contains the more difficult implication of Theorem 4.1 which is a 
consequence of the proposition below. 

Proposition 4.4. Fix a morphism a : — > (H, V), with (H, V) satisfying the A2 identity. 
For each h G H, the set of forests t with type a(t) = h is definable by a S2 forest 
expression, and thus also by a formula 0/ S2(<, <i ex ). 



Before proving this proposition, we show how it concludes the proof of Theorem 4.1 
Since S2 expressions allow union, the above proposition shows that any language recognized 
by a can be defined by a £2 forest expression. In particular, if L is recognized by a, then 
both L and its complement can be defined by S2 forest expressions, and consequently 
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formulas of £2(<> <iex)- Since the complement of a £2(<,<icx) formula is a Il2(<, <i c 
formula, we get the right-to-left implication in Theorem |4.1| 



The rest of the paper is devoted to showing Proposition 4.4 The proof is by induction 
on two parameters. For the second parameter, we need to define a pre-order on H. We say 
that a type h is reachable from a type g if there is a context type v G V such that h = vg. 
If h and g are mutually reachable from each other, then we write h ~ g. Note that ~ is an 
equivalence relation. Note also that if g is reachable from h, then h is a piece of g. We write 
H± for the set of types h that can be reached from every type g S H. Note that H± is not 
empty, since it contains the type hi -) + h n , for any enumeration H = {hi, . . . , h n }. 



The proof of Proposition 4.4 is by induction on the size of the algebra (H, V) and 
then on the position of h in the reachability pre-order. The two parameters are ordered 
lexicographically, the most important parameter being the size of the algebra. That is we 
will either decrease the size of the algebra or stay within the same algebra, but go from 
a type g to a type h such that h is reachable from g but not vice versa. As far as h is 
concerned, the induction corresponds to a bottom-up pass, where types close to the leaves 
are treated first. 

Part of the induction proof is presented in Section [5] However, the induction breaks 
down for types from H±, which are treated in Section [7j 

5. Types outside Hi 



In this section we prove Proposition 4.4 for forest types outside H±. We fix such a 



forest type h for the rest of the section. By induction assumption, for each type g ^ h from 
which h is reachable, we have a £2 forest expression defining the language L g of forests of 
type g. (The case when there are no such types g corresponds to the induction base, which 
is treated the same way as the induction step.) In this section we assume that h is outside 
H±, and we will produce a £2 forest expression for L^. The case where h is in H± will be 
treated in Section 

In the following, we will be using the stabilizer of h, defined as 

stab(h) = {v : vh ~ h} C V . 

We say that a context type v stabilizes h if it belongs to the stabilizer of h. The key lemma 
is that the A2 identity implies that the stabilizer is a submonoid of V. 

Lemma 5.1. The stabilizer of h only depends on the ^-class of h. In particular, it is a 
submonoid of V . 

Proof. We need to show that if h ~ h! then stab(h) = stab(h'). Assume v £ stab{h). Then 
vh ~ h. Hence we have u\, ui, u% such that h = u±vh, h = i^/i' and h! = U3J1. This implies 
that h' = u^u\vu2h' and therefore h' = {u^u\vu2) u) h' . From the A2 identity we have that 

h! = (U3U1V U2) UJ h / = (UZU1VU2Y v{u^U\VU2) U3 h' = (U3U1V U2) w vh f . 

Hence h! is reachable from vh! . Since vh! is clearly reachable from h! , we get vh! ~ h! and 
v £ stab(h'). 

To see that stab(h) is a submonoid consider v,v' £ stab(h). We need to show that 
vv' £ stab{h). Let h' = v'h. Because v' G stab{h) we have h! ~ h. As v 6 stab(h) = stab(h') 
we have vh' ~ h! and hence vv'h ~ h. □ 
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Recall now the piece order on forest types from the end of Section 2.2 which corresponds 
to removing nodes from a forest. We say a set F C H of forest types is closed under pieces 
if any piece of a forest type / G F also belongs to F. A similar definition is also given for 
sets of context types. Another consequence of the A2 identity is: 

Lemma 5.2. Each stabilizer is closed under pieces. 

Proof. We need to show that if u stabilizes h, then each piece u' of u also stabilizes h. By 
definition of the stabilizer we have a context type v such that h = vuh. We are looking for 
a context type w such that wu'h = h. Prom h = vuh we get h = (vu) u h. Hence by the A2 
identity we have h = (vu) u u' {vu) u h = (vu) u u'h as desired. D 

We now consider two possible cases: either h + h ~ h, or not. Equivalently, we could 
have asked if h + □ stabilizes h. Again equivalently, we could have asked if □ + h stabilizes 
h. When h + h ~ h, we will conclude by induction on the size of the algebra. When 
h + h 7^ h, we will conclude by induction on the reachability pre-order. These cases are 



treated separately in Sections 5.2 and 5.1, respectively. 



5.1. h + h ~ h. Let G be the set of pieces of h. By assumption that h + h ~ h, we know 
that both h + □ and □ + h stabilize h. 

Lemma 5.3. If h + h ~ h then {G, stab(h)) is a forest algebra. 
Proof. We need to show that the two sets are closed under all operations: 

stab(h)stab(h) C stab(h) 

G + G C G 
□ + G,G + D C stab(h) 
stab{h)G C G 



The first of the above inclusions follows from Lemma 5.1 For the second inclusion, we note 
that h + h is a piece of /i by assumption on h + /1 ~ h. In particular, each forest type in 
G + G is a piece of /i. For the third inclusion, h + /i ~ /i implies that both /i + □ and 



□ + h stabilize h. Since by Lemma 5.2 the stabilizer is closed under pieces, we get the third 
inclusion. For the last inclusion, consider v £ stab{h) and g £ G. We need to show that 
vg £ G. This holds because vg is a piece of vh, which is a piece of h as vh ~ h. D 

Recall that in this section we are dealing with the case when h is outside H±, i.e. there 
are some forest types that can be reached from h but not vice versa. In this case we show 
that G is a proper subset of H. To see this, we show that for h± £ H±, h± G. Assume 
for contradiction that h± G G. Then h± is a piece of h and because h + □ G stab(h) and 



stab(h) is closed under pieces (Lemma 5.2), we infer hj_ + □ G stab(h) and /i is reachable 
from a contradiction. 

Therefore the algebra from the above lemma is a proper subalgebra of the original 
(H, V). Furthermore, this algebra contains all pieces of h; so it still recognizes the language 
Lh\ at least as long as the alphabet in the morphism is reduced to include only letters that 
can appear in h. We can then use the induction assumption on the smaller algebra to get 
a £2 forest expression for L^. 
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5.2. h + h 7^ h. For v G V, we write iT^ for the set of contexts of type v. For g G H, we 
write L g for the set of forests of type g, and M 5 for the set of trees of type g. 

Let G be the set of forest types g such that h is reachable from g but not vice- versa. By 
induction assumption, for each g G G, the language L g is definable by a £2 forest expression. 
Our goal is to give a £2 forest expression for L^. 

Lemma 5.4. .Am/ forest t of type h can be decomposed as t = ps, with s a forest whose type 
is reachable from h, and which furthermore is: 

(1) A tree s = as' with the type of s' in G; or 

(2) A forest s = si + S2 with the types of s%, S2 in G. 

Proof. Consider decompositions of t as t = ps. Among such decompositions, take a decom- 
position where the forest s has a type reachable from h, but s has no subforest with a type 
reachable from h. Such a decomposition always exists as t is of type h. If s is a tree, we 
get case (1), if s is a forest, we get case (2). □ 

Note that in the above lemma, the type of the context p must stabilize h, since both 
the type of s and the type of the whole forest t are in the class of h. Therefore, thanks to 
the above lemma, the set Lh of forests with type h can be decomposed as 

Lh= (J ( U K u aL g U |J K u (L gi +L 92 )) (5.1) 

u&stab(h) a€A,g£G 3i,92SG 

uag=h u(g\+g2)=h 

Note that L g , L gi and L g2 can all be written as £2 forest expressions thanks to the induction 
assumption from Proposition |4.4| The only thing that remains is showing that the context 
language K u can be defined by a £2 context expression. For this, we use the following 
proposition. 

Proposition 5.5. For any v G V , the language K v of contexts of type v is defined by a 
finite union of concatenations of the form K\ ■ ■ ■ K m where each context language Ki is 
either: 

(1) A singleton language {ad} for some a G A; or 

(2) A context language closed under pieces; or 

(3) A context language □ + M g or M g + □ for some g G H. 

The proof of this proposition will be presented in Section |6j Meanwhile, we show how 



the proposition gives a £2 context expression for each language K u in (5.1). The singleton 
languages, and the languages closed under pieces are £2 context expressions by definition. 
The only potential problem is with the languages □ + M g or M g + □ that appear in the 



proposition. Since the context types u that appear in (5.1 ) stabilize h, the forest type g has 
to be such that □ + g or g + □ stabilizes h. In either case, g cannot be reachable from h, 
since h + h ^ h and the stabilizer is closed under pieces. As h is obviously reachable from g, 
the language L g is definable by a £2 forest expression thanks to the induction assumption 



from Proposition 4.4 Finally, M g is the intersection of L g with the set of all trees, which 



is definable by a £2 forest expression 
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6. TREATING CONTEXTS LIKE WORDS 



In this section, we prove Proposition 5.5 The basic idea is that a context is treated 



as a word, whose letters are smaller contexts. The proof strategy is as follows. First, in 
Section |oTT} we present the characterization of A2(<) for words, which was shown by Pin and 
Weil in [11] . This characterization is slightly strengthened to include what we call stratified 



monoids, which are used to model the contexts that appear in Proposition 5.5 Then, in 



Section 6.2, we apply the word result, in its strengthened form, to prove Proposition 5.5 



6.1. A2(<) for words. In this section we present the characterization of A2(<) for words, 
extended to stratified monoids. A stratified monoid is a monoid M along with a pre-order 
^ that satisfies the following property: 

m nm = m for n <m . 

A subset iV C M is called downward closed under < if for every n £ N, and every m < n, 
we also have m 6 N. 

Proposition 6.1. Let A be an alphabet (possibly infinite), and let (3 : A* — > M be a 

morphism into a stratified monoid (M, ^) that satisfies the identity 

(mn) 11 m(mn) w = (mn) u (6-1) 
For any m E M, the language (3~ 1 {m) is defined by a finite union of expressions 

A^Al ■ ■ ■ BiA* 

where each Bj is of the form A n /3 _1 (n) for some n £ M, and each Aj is of the form 
A n /3 -1 (iV) for some N C M downward closed under ■<. 

The difference between the above result and the main technical result in Pin and Weil 
is twofold. First, we use infinite alphabets here. Second, we use stratified monoids to 
get a stronger conclusion, where the letters in the blocks A* are downward closed. Both 
differences are necessary for our application to context languages. 

Our proof is a straightforward adaptation of a proof of Therien and Wilke in |16j . which 
analyzed the languages recognized by semigroups in DA. 



Before proving this result, we remark how Proposition 6.1 gives the characterization of 
A2(<) presented by Pin and Weil: 

Corollary 6.2 (Pin and Weil |llj). A word language (over a finite alphabet) is definable 



A2(<) if and only if its syntactic monoid satisfies the identity (6.1). 



Proof. The only if implication is shown using a standard Ehrenfeucht-Fra'isse argument, we 
only consider the if implication. 

Let then L C A* be a language recognized by a morphism (5 : A* — > M, with M 



satisfying (6.1). We can see this M as a stratified monoid under the identity pre-order. By 



applying Proposition 6.1 we see that each inverse image (5{m) is defined by an expression as 



in Proposition 6.1 (the downward closure is a vacuous condition, since the order is trivial). 



Since each such expression is clearly expressible in Sa(<), we get that L is definable in 



E2(<). Furthermore, by Proposition 6.1 also the complement of L is definable in S2(<), 



and therefore L is also definable in Il2(<). D 
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We now proceed to the proof of Proposition 6.1 The proof is by induction on the size 
of /3(A) C M or, equivalently, the number of elements in the monoid that correspond to 
single letters. In the proof of Therien and Wilke, the induction was simply on the size of 
the alphabet, but this will not work here, since the alphabet is infinite. 

We use the term £2 word expression for the word expressions as in the statement of the 



Proposition 6.1 It is not difficult to show that languages defined by £2 word expressions 
are closed under union, intersection and concatenation. 

We will use the following notation. Given two elements m and n of M we say that 
m ~£ n if there exist k,l G M such that m = kn and n = Im. We say that m n if 
there exist k,l G M such that m = nk and n = ml. These are the left and right Green's 
relations. 



A classical consequence of aperiodicity, itself a consequence of (6.1), is: 



m n A m ~£ n m = n for m,n G M . (6-2) 



We will also use the following property of monoids satisfying (6.1 ), which can be proved 



along the lines of Lemma 5.1 



m ~7£ n ~7£ mk nk n for m,n,k G M . (6-3) 

Lemma 6.3. For all m G M, the language U m = {w : m(3(w) m} is definable by a S2 
word expression. 

Proof. Let A m be the set of letters a of A such that m/3(a) ~7£ m. In other words, A m = 
A n (3^ 1 (N), where ./V is the set {n : mn m}. 

We will show that U m = A* m . Stated differently, a word belongs U m if and only if all of 
its letters belong to A m . 



Thanks to (6.3), for all n m we have n(3(a) m. Hence by induction on the 
length of w G A* m we can prove w G U m . For the converse implication, let w be a word 
outside A* m , of the form w = w\aw2 with w\ G A* m and a A m . Then m/3(w) = nf3(a)n' 



for some n,n', and from the discussion above we have n m. Hence by (6.3) and by 
hypothesis on a, nf3(a) m and w cannot be in U m . 

To conclude, we need to show that N is closed under X. Indeed, let k ■< n and let 
n G N. By assumption on the monoid being stratified, we have n^knf = n w . In particular, 
we have 

mnfkn^ mn^ m . 



From the above it follows that mn^k m, which gives mk m by (6.3), and hence 



k G JV. □ 



Note that by (6.3) we have U n = U m whenever m n. 



Lemma 6.4. For any m G M , the following language can be defined by a £2 word expres- 
sion: 

V m = {at ■ ■ ■ a,i : (3(ai ■■■a i ) = m and fi(a\ ■ ■ ■ Oj_i) m} . 



Before we give the proof, we show how it concludes the proof of Proposition 6.1 Con- 
sider the set {w : [3(w) m}. Each word w in this set can be written as uv where u is 
the smallest prefix of w such that (3(u) m. Hence we have: 

{w : (3(w) m} = [J V n U m . 



TREE LANGUAGES DEFINED IN FIRST-ORDER LOGIC WITH ONE QUANTIFIER ALTERNATION 15 



Since £2 word expressions are closed under union and concatenation, the above language 
is definable by a £2 word expression than 
version of Lemma 6.3 and Lemma 6.4 for 
/3(w) ~£ m}. But we also know from (6.2) that 



is definable by a £2 word expression thanks to Lemmas 6.3 and 6.4 Using a symmetric 

~£, we can get an £2 word expression for {w : 



(3 (m) = {w : /3(w) m} n {w : (3(w) ~£ m} 
and the result follows by closure of £2 word expressions under intersection. 



Proof of Lemma 6.4- We say that mis a prefix of n if there exists k G M such that n = mfc. 
This defines a pre-order in M. The proof is by induction on the position of m relative to 
this pre-order. 

The induction base is when m has no proper prefixes: If m = nk then n m. In this 
case the language V m contains at most the empty word, since the condition on a% ■ ■ ■ a\—\ 
is infeasible. Clearly both languages and {e} are £2 word expressions. 

Assume now that m is not minimal. Each word w of V m can be written as ua where 
a £ A, (3(u) = k and k(3(a) = m. Furthermore, u can be written as u\U2 where u\ is the 
smallest prefix of u such that n = (3(ux) k. We therefore have: 

Vm = [J V n U n:k a with U n>k = {w £ A* : nf3(w) = k} 

where the union is taken for n,k £ M such that n k, k 9^7^ m a prefix of m and for 
a £ A with kf3(a) = m. By induction the language V n is definable by a £2 word expression. 
It is also clear that U n ^ Q U n . Recall from the proof of Lemma 6.3 that U n = A* n where 
A n = An^^JV) for some N O M. Therefore we also have /3(f7 n ,fc) Q A* n . From ( 6.3 1 and 
the fact that kf3(a) = m we know that a A n , therefore A n is a proper subset of A. Let (3' 
be the restriction of /3 to A n . We have U n ^ = Unx=fc f rom induction on the size of 

This concludes the 
□ 



the alphabet in Proposition 6.1 we obtain a £2 word expression for U n ^ 
proof of this lemma as £2 word expressions are closed under concatenation and union. 



6.2. Proof of Proposition 5.5 , We now proceed to show how the word result stated in 



Proposition 6.1 can be lifted to the context result in Proposition 5.5 Proposition 5.5 says 
that for any context type v G V, the set of contexts of type v is described by a finite union 
of expressions of the form K\ ■ ■ ■ K m where each K{ is either: a singleton language {ad}; 
or closed under pieces; or an expression □ + M g or M g + □. 

The basic idea is that we treat the context as a word over an infinite alphabet, which 
we call B. This alphabet has two kinds of letters. Both kinds are contexts: 

• Contexts of the form ad, for a £ A. 

• Contexts of the form t + □ or □ + 1, for t a tree over A. 

Consider now the morphism (3 : B* — >■ V, which is simply a restricted to the contexts 
in B* . Every context p in K v can be decomposed as p = b% ■ ■ ■ b m G B*. In particular, we 
have 

K v = p~\v) . 

We can treat V as a stra tified monoid, by using the piece relation H as the pre-order. 

we see that the inverse image fi~ l {v) can be presented as a 



6.1 



By applying Proposition 
finite union of expressions of the form: 

(sn/r'Wnsn/rVi))' 



(Br\p-\nk)){Bc\p-\N k )y 
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where n\, . . . , n& are elements of V, and JVi, . . . , iV^ are a subsets of V that are downward 
closed under X. 

We need to show that the expressions used above are of the three forms allowed by 
Consider first an expression (B n f3~ 1 (W))* , where W C V is closed under 



5.5 



Proposition 

pieces. Since TV is closed under pieces (as a set of context types), then so is the language 
(B n (3~ l (W))* (as a set of contexts). Consider next an expression of the form B n /3 _1 (n). 
This context language is a union of languages of the first (singleton) and third (□ + M g 

The union is not a problem for a £2 word 



5.5 



or M g + □) types described in Proposition 
expression, since union distributes across concatenation. 



7. Types in Hj_ 

Recall that in Section [5} we managed to find a £2 forest expression for each set L^, 
assuming h was outside H±. Our techniques failed for forest types h G H±, i.e. forest types 
reachable from every other forest type. In this section, we deal with these forest types. 

In order to deal with the types from H±, we will have to do a different induction, this 



time on context types. This induction, stated in Proposition 7.1, is expressed in terms of 
an equivalence relation =„. Given a context type v € V and two forest types h,h', we write 

h = v h if Vu G V vuh = vuh' . (7.1) 

We extend this equivalence relation to context types, by 

w = v w if \/u G V V/j G H vuwh = vuw'h . (7-2) 

By abuse of notation, we also lift the equivalence relation = v to forests, considering two 
forests s, t equivalent when their forest types are equivalent. It is this meaning that is used 
in the statement below. 

Proposition 7.1. For any context type v, every equivalence class of forests under = v is 
forest language definable by a £2 forest expression. 



From Proposition 7.1 we immediately obtain a £2 forest expression for L/j, as the 



equivalence class of = v containing h, where v = □. Hence the proof of Proposition 7.1 ends 



the proof of Proposition 4.4. We note that the proof of Proposition 7.1 will be using the £2 
forest expressions for types h H± that have been developed in Section [5j In particular 
if an equivalence class of = v is contained in H \ H± , then it can easily be defined by the 
disjunction of all the £2 forest expressions corresponding to each type. The difficulty is to 
handle equivalence classes that intersect H±. 



The rest of Section[7|is devoted to proving Proposition 7.1 The proof uses the following 



pre-order on context types. We say that a context type u is a prefix of a context type v 
if there exists a context type w such that v = uw (we also say that v is an extension of 
u). We overload the use of ~ and denote by ~ the equivalence relation induced by the 
prefix pre-order, i.e. v ~ w holds if v is both a prefix and an extension of w. The proof of 



Proposition 7.1 is by induction on the position of v in the prefix pre-order, starting with 
context types that have no proper extension, and ending at the context type v = □ that 
has no proper prefix. 
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7.1. The induction base. The base of the induction in the proof of Proposition 7.1 is 
when the context type v has no proper extension, i.e. v ~ w holds for all extensions w of 
v. We will show that such a context type is necessarily constant, i.e. vg = vh holds for 
all forest types g,h € H. This gives the induction base, since for a constant context type 
v, there is only one equivalence class of =„, and this class is, by definition, the set of all 
forests, which can be defined by a £2 forest expression (it is closed under pieces). 

Lemma 7.2. A context type has no proper extension if and only if it is constant. 

The if direction is immediate: if a context type v is constant, then vu = v holds for all 
context types u, and therefore v has no proper extension. For the converse implication, as 
well as in the rest of Section [7j we will use the notion of stabilizers for context types: 

stab(v) = {w : vw ~ v} . 

When u S stab(v), we say that u stabilizes v. As for stabilizers of forest types (recall 



Lemma 5.1 and Lemma 5.2), the A2 identity implies that the stabilizer stab(v) is a sub- 
monoid of V and it is closed under pieces. The following lemma implies Lemma |7.2[ since 
its assumptions are met by a context type without proper extensions. Recall that H± is 
the equivalence class of ~ that contains all types reachable from any other type. 

Lemma 7.3. If both H± + □ and □ + H± intersect stab(v), then the context type v is 
constant. 

Proof. First note that if some context type in H± + □ stabilizes v, then all context types 
in H± + □ stabilize v, likewise for □ + H±. This is because the stabilizer is closed under 
pieces, and every type in H + □ is a piece of every type in H± + □. 

Let / = hi + • • • + h n , for some arbitrary enumeration hi, ■ ■ ■ , h n of H. As we noted 
above, both / + □ and □ + / stabilize v. Therefore, 

v(uf + □ + oof) ~ v . 

The context type (ujf + □ + oof) is constant, since any forest type h G H is a piece of /, 



and therefore by (4.1) we have 

Ojf + h + Uf =(/ + □)"• (h + □).(/ + □)«. = (/ + nr • (/ + □)" • = Oof + Uf . 

Hence the context type v is constant as it is equal to wv{ujf + □ + ojf) for some context 
type w. □ 



7.2. The induction step. We now proceed to the induction step in Proposition 7.1 Recall 
that our goal is to find a £2 forest expression for every equivalence class of = v . For a forest 
language L, we denote by \L\ V the union of equivalence classes of =„ that intersect L, i.e. 

[L] v = {t : t = v s for some s £ L} D L . 

We use a similar notation [K] v for languages of contexts. We say that a forest language is 
a v- over approximation of a forest language L if it contains L, but is contained in [L] v . In 
other words, a w-overapproximation may add forests to L, but it adds no new forest types, 
at least as far as the context type v is concerned. Note that a language may have several 
v-overapproximations . 

Proposition 7.4. For every h £ H, some v- over approximation of Lh can be defined by a 
£2 forest expression. 
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The above result concludes the proof of Proposition 7.1 To see this, consider an 
equivalence class consisting of forest types /i, • • ■ , f n - The set of forests with a type in the 
class is by definition equal to [JLf.. By definition of v-overapproximation this set is also 
equal to [JLf. where Lf. is any u-overapproximation of Lt v Hence it is definable by a £2 



forest expression by Proposition 7.4 



The rest of this section is devoted to proving Proposition 7.4 

When h is outside H±, then itself, which is its own v-overapproximation, is definable 
by a £2 forest expression by the results from the previous sections. The problem is when 



h is in H±. Because v has some proper extension, from Lemma 7.3 we know that at least 
one of □ + H± or H± + □ is disjoint with the stabilizer of v. Without loss of generality we 
assume 

n + H ± n stab(v) = . (7.3) 
In other words, if the type of a context stabilizes v , then it is possible that some tree to the 
left of the hole has a type in H±, however all trees to the right of the hole must have types 
outside H±. 

In order to obtain a w-overapproximation of for h E H± we will use the following 
decomposition of forests with types in H± . 

Lemma 7.5. Any forest t of type in H± has a decomposition t = ps where p is a context 
whose type stabilizes v, and s is a forest of type in H± that has one of the two forms below. 

(1) s = as' with the type of the forest s' outside H±; or 

(2) s = as' with the type of the context a\3 not stabilizing v; or 

(3) s = si + S2 with the types of the forests s\, S2 outside H±; or 

(4) s = si + S2 and 

• if si has type in H±, then the type <?/□ + S2 does not stabilize v. 

• if S2 has type in H±, then the type of s\ + □ does not stabilize v. 

Proof. Consider the set D of all possible pairs (p, s) such that t = ps, the type of p preserves 
v and the type of s is in H± . Take a pair (p, s) £ D that is maximal in the following sense: 
if q is a nonempty context, then D has no pair with pq on the first coordinate. 

Suppose s is a tree of the form s = as'. If s' has a type outside H^, we get item ([!]). If 
s' has type in H±, then by maximality, the context type of paO does not stabilize v, and 
therefore aO has a type that does not stabilize v, so we get item ([2]). 

Suppose s is a forest of at least two trees. Consider any partition of s into two nonempty 
forests s = s\ + S2- If both s±, S2 have type outside H±, then we get item ([3]). Otherwise, 
we get case Q by maximality of (p, s). □ 

From Lemma 17.51 we have for h S H±: 



u£stab(v) feH± 
uf=h 



where Yf stands for the set of all forests s that have type / £ H± and that satisfy one 
of the conditions Q-Q of Lemma 7.5 To get the u-overapproximation of we will 



use v-overapproximations for the smaller expressions above, as stated by the following two 
lemmas. The first lemma is concerned with languages of the form Yf. 

Lemma 7.6. For every f 6 H±, some v- over approximation Yf ofYf can be defined by a 
£2 forest expression. 
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For the second lemma, concerning K u , we need a more careful statement. The overap- 
proximation that we give is not really an over approximation of K u , but it is an overapprox- 
imation that works as long as a forest of type in H± is inserted into the hole. 

Lemma 7.7. For any u G stab(v), one can define a £2 context expression K u such that 
for any forest s of type in H±, K u s is a v-overapproximation of K u s. 



These two lemmas are proved in Sections 7.2.2 and 7.2.1 respectively. First we show 
how they complete the proof of Proposition 7.4 We write L± for the set of all forests with 
a type in H±. We claim that the following language 

L h = (J |J K u -(Y f HL ± ) 

u£stab(v) fEH± 
uf=h 

is a f-overapproximation of L^. 

The first property required from a w-overapproximation, Lh Q L\, is immediate. For 
the second part, Lh C [Lh] v , we need a bit more effort. We show a stronger result, namely 
that for any u and / as in the summation above, we have 

K u (Y f nL ± ) C [K u Y f ] v 

This completes the proof of Lh C [Lh] v , since [_]„ distributes across union. To prove the 
above, we apply the properties of K u and Yf to get 

K u (Y f nL ± ) C [K u ■ (Y f n L ± )] v C [K u -Y f ] v C [K u ■ [Y f ] v ] v 

To complete the proof, we would like to replace [Yf] v by Yf in the last expression above. 
This can be done thanks to the following easily verifiable consequence of the fact that = v 
is a congruence for forest algebras. 

Fact 7.8. For any set of contexts K and set of languages L, we have = [KL] V . 



Thanks to Lemmas 



7.7 



and 



7.6 



the only thing keeping from being defined by a £2 



forest expression is the language L±. We deal with this language in the following lemma. 

Lemma 7.9. The language L± is definable by a £2 forest expression. 

Proof. A subforest of t is a forest s such that t = ps for some context p. Take a forest 
t E Lj_ and consider a subforest s of t that is in L±, but has no proper subforests in L±. 
Then either s is a tree as' with s' $ L±, or s = s\ + S2 with s\, S2 L±. Therefore, a forest 
is in L± if and only if it has a subforest in 

[J aL g U [J L gi + L 92 . 

a£A,g£H ± 91,92&H ± 
ag<=H ± 9i+92GH ± 

Containing such a subforest can be expressed by a £2 forest expression, by prefixing the 
set above with the set of all contexts. The expressions for L g ,L gi and L g2 are £2 forest 
expressions by the results from the section on types outside H±. □ 
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7.2.1. Proof of Lemma 7. 7 Our goal in this section is to prove Lemma 7.7 which says that 
for any context type u stabilizing v, there is a T,2 context expression K u such that for any 
forest s with a type in H±, K u s is a v-overapproximation of K u s. 

We apply Proposition 5.5 to get an expression for the context language K u of the form 



■ ■ K, 



(7.4) 



The problem with the expression above is that it may use, in some of the subexpressions 
Kij, languages Mf + □ or □ + Mf that involve forest types / E H±, and we do not know 
how to describe types in H±. This is where the overapproximation comes in. We show 
that if the languages for types in H± are overapproximated, then the result satisfies the 
properties required by Lemma 7.7 A more detailed argument is described below. 

We say a context language K satisfies (*) if it has the property required from K u by 
Lemma |7,7[ namely 

(*) there is a £2 context expression K such that for any forest s with a type 
in H±, the language Ks is a w-overapproximation of Ks. 

It is not difficult to see that property (*) is preserved by unions and compositions of context 
languages. In particular, in order to prove Lemma 7.7, it suffices to show (*) is satisfied by 
all languages K^j that appear in (7.4). 

The only problem with the overapproximation is when Kij is of the form Mf + □ or 
□ + Mf, for / G H±. In all the other cases, Kij is known to be definable by a £2 context 
expression, and no overapproximation is needed. Note that by (7.3), the expressions d + Lf 
cannot be used, since a forest type from H± cannot appear to the right of the hole in a 
context type that stabilizes v. Therefore, to complete the proof of Lemma 7.7, it remains 
to show that for any / G H±, the context language Mf + □ satisfies (*). 

In the following, we use an equivalence relation = v +. This is defined to be the intersec- 
tion of all equivalence relations = vu , for context types u that do not stabilize v (and hence 
v is a strict prefix of vu). For a forest language L, we write 

[L] v+ = p| [L] vu (7.5) 

u$_stab(v) 

By the induction assumption in Proposition 7.1, each equivalence class of = v + is definable 
by a T,2 forest expression and therefore so is each language as a union of equivalence 

classes of =„+. We will show that, for any / G H±, the context language K = Mf + □ 
satisfies (*) with K = |X/]t>+ + D. Assume that the type of s is in H±. Then we have: 

Ks = M f + s C Ks = [L f ] v+ + s C [Ks} v = [L f + s] v . 

The first inequality is clear. For the second inequality, we need to show that 

v ■ a([L f] v+ + s) C v-a(Lf + s) 

This inclusion holds because we have: 

v ■ a([Lf] v+ + s) = v ■ (□ + a(s)) ■ a([Lf] v+ ) =«•(□ + «(s)) • a(Lf) = v ■ a(Lf + s) 

In the second equality, we used the definition of [£/]„-(- and the assumption that □ + a(s) 
does not stabilize v. The latter follows from assumption (7.3) since the type of s is in H±. 
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7.2.2. Proof of Lemma 7.6. In this section, we show that for every type / £ H±, a v- 
overapproximation of Yt can be defined by a £2 forest expression. Recall that the language 
Yf was defined based on a case distinction in Lemma 7.5, and therefore it can be decom- 
posed into a union of four languages, one for each of the four cases in the lemma. As 
v-overapproximations are closed under union, for each of these languages, we provide a 
v-overapproximation defined by a £2 forest expression. 

For the languages corresponding to cases @ and (§, we use the assumption that Lh 
can be defined by a £2 expression for every type h H±. The interesting cases are ((2]) 
and Q. 

The language corresponding to case ([2]) is a union of forest languages of the form 

a ■ L h (7.6) 

ranging over letters a such that aD does not stabilize v, and forest types h with ah = f. 
We treat each language a ■ Lh separately. 

It may be the case that h belongs to H± and therefore we have no £2 forest expression 
for Lh. However, we can use overapproximation. As a does not stabilize v, we can apply 
the induction assumption in Proposition |7.1| and obtain a £2 forest expression for 
But then the £2 forest expression a ■ [Lh] v + is a u-overapproximation of a ■ Lh- It clearly 
contain a ■ Lh so it remains to show that it is included in [aL/J„. To see this consider a 
forest t £ [Lh] v + of type g and an arbitrary u £ V. As a does not stabilize v, ua does not 
stabilize v. Hence by the choice of g we have vuag = vuah and at is in [aLh} v - 

It remains to consider the case of Q, where have a union of sets 

Lhi + L h2 (7.7) 

ranging over h\, hi that satisfy the two implications in item Q of Lemma 7.5 We do each 
pair hi, hi separately. We consider three subcases. 

The first subcase is when h\ H±. Therefore we have a £2 forest expression for Lh x - 
In this case we claim that 

^ + \ L h 2 ]v+ 

is a ^-overapproximation of of L/ ll + Lh 2 ■ The first requirement of v-overapproximation, 

+ Lh 2 Q L hl + {L h2 ]v+ , 

is immediate. For the other requirement, we need to show that for any forests t\ E Lh x 
and *2 £ [Lh 2 ]v+, the type of t\ + ti is =t,-equivalent to h\ + hi- From t\ £ L^, we know 
that the type of t\ is hi, but all we know about ti is that its type gi satisfies gi = 1I+ hi. 
Consider an arbitrary u S V. Since hi + □ does not stabilize v, we also have u{h\ + □) 
does not stabilize v. Hence from gi = v+ hi we get vu(hi + gi) = vu(h\ + hi). 

The second subcase, when hi H±, is treated as above by symmetry. 

The third subcase is when both hi and hi are in H±. As a consequence of hi G H± 
and the second condition of item Q in Lemma |7.5| is that 

H ± + □ n stab(v) = . (7.8) 

We claim that a -u-overapproximation of + L^ 2 is 

([L hl ] v +nL ± ) + ([L h2 } v+ nL ± ) . (7.9) 

As before, the problem boils down to showing that for any forests 

t\ £ [-kftJtH- nil t 2 £ [Lh 2 ] v + n L± , 
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the types g\ of t\ and g 2 of t 2 satisfy g\ + g 2 =v hi + hi- In other words, we need to show 
that for an arbitrary u G V 

vu(gt + g 2 ) = vu(hi + h 2 ) . 
Since t\ G Lj_, then by (7.8) the context type 51 + □ does not stabilize t> and therefore the 
same holds for u{gi + □). Hence, we can use the assumption on g 2 =v+ h 2 to infer 

vu(gi + g 2 ) = vu(gi + U)g 2 = vu{g\ + U)h 2 = vu{g\ + h 2 ) . 

In a similar way, we use h 2 G H±, the assumption ( |7.3[ ), and <?i =„+ /ii, to complete the 
proof of this case, and of Lemma |7.6| 

vu(gi + h 2 ) = vu(D + h 2 )gi = vu(D + h 2 )hi = vu(h\ + h 2 ) . 



8. NO LEXICOGRAPHIC ORDER 

In this section, we consider the logic A2(<) where only the descendant order, and not 
the lexicographic order, is available. We give an effective characterization in the following 
theorem. 

Theorem 8.1 (Effective characterization of A2 with the descendant order only). 

A forest language is definable in A2(<) if and only if its syntactic forest algebra satisfies 

the A 2 identity, as well as horizontal commutativity: 

h + g = g + h. (8.1) 

The "only if" implication is easy: we have already shown that the A2 identity must 
hold in the syntactic forest algebra of a language definable in A2(<,<i cx )5 and A2(<) is 
a fragment of A2(<,<i cx )- Horizontal commutativity must also hold: the logic only has 
the descendant relation, and therefore its formulas are invariant under rearranging sibling 
subtrees. 

The "if" implication is a minor variation on the work done in the previous sections. 
Recall that we proved before that if the syntactic forest algebra of a language L satisfies 
the A2 identity, then both L and its complement can be defined by T, 2 forest expressions. 
We apply this result also in our case. The problem is that the T, 2 forest expressions are not 
commutative, and thus need not be definable in S2(<). We will show, however, that their 
commutative closure can be defined in S2(<). Here, we use the term commutative closure of 
L for the smallest language that contains L and is closed under rearranging sibling subtrees. 

Proposition 8.2. The commutative closure of a T, 2 forest expression is definable in £2(<). 

Before we prove this proposition, we remark that this is not a completely generic result. 
For instance consider the following language over a one letter alphabet: "Each node is a leaf 
or has two children, and some leaf has an even number of ancestors" . This language is defin- 
able in £3(<j <i ex ) and is horizontally commutative (the formula comes from Potthoff |12j). 
However, this language cannot be defined in E3(<). Actually, an Ehrenfeucht-Fraisse ar- 
gument shows that every first-order formula, that has quantifier depth n and only uses the 
descendant order, will give the same result for all balanced binary trees of depths larger 
than 2 n . 

The proof of the above proposition is by induction on the size of the £2 forest expression. 
The base case is when the £2 expression is either {ad}, or a language that is closed 
under pieces. In the first case, the language is clearly definable in S2(<). In the second case, 
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we revisit the proof of Lemma 3.1 which showed that a language L that is closed under 
pieces is definable in LTi. If we take the commutative closure of L, we get a commutative 
language closed under pieces. In the proof of Lemma 3.1 we constructed the IIi formula 
by forbidding a finite number of pieces; in the commutative case the formula does not need 
to worry about the order of siblings in the forbidden pieces. 

In the induction step, we have to consider the operations that are allowed by £2 ex- 
pressions: union, intersection, (horizontal) concatenation 

L + U (8.2) 

and (vertical) composition 

K ■ L K-K' (8.3) 
for a forest languages L,L' and context languages K,K'. Union and intersection are easy. 
Concatenation and composition are more problematic. Actually, ^2(<) is not closed under 
these two operations. For instance, the languages 

K = {a + □,□ + 0} 

L = {b + c, c + b} 

are both definable in E2(<), but their concatenation 

KL = {a + 6 + c, a + c + b,b + c + a,c + b + a} 

is not, since it does not contain the forest b + a + c. 

Nevertheless, if we use commutative closure, the problem disappears. That is, we will 
show that if the languages K, K' , L, L' are definable in S2(<), then the commutative closure 
of each of the languages in (8.2) and (8.3) can be defined in ^2(<). We only do the cases 
L + L' and K-K', the language K ■ L is done the same way. 

Lemma 8.3. If forest languages L,L' are definable in £2(<), then so is the commutative 
closure of L + L' . 

Proof. We write L © V for the commutative closure of L + L' . 

Consider first the case when V is a tree language definable in S2(<). In this case, the 
formula for L © L' formula places an existentially quantified variable over the root of one 
tree, and then relativizes the formulas for L and U , respectively, to the nodes the are not 
descendants (respectively, are descendants), of this existentially quantified root. 

For the general case, we use the following lemma on forest languages definable in E2(<). 

Lemma 8.4. Every forest language L definable in £2(<) can be written as a finite union 
of languages Lq © Mi © • • • © M n , where Lq is a forest language definable in LIi(<) ; and 
Mi, . . . , M n are tree languages definable in S2(<). 

Proof. The statement of the lemma immediately follows from the following claim on formu- 
las of S2(<). We claim that any formula <p of ^2(<) is equivalent to a finite disjunction of 
formulas of the form 

3zi . . . 3z n ip A f\ (Vy ->(y < zfi) A fa 

ie{l,...,n} 

where ip G ili(<), and fa, . . . , ip n & ^2(<) are formulas such that 

• Each formula fa has all quantification relativized to descendants of z- t . 

• The formula if) has all quantification relativized to nodes that are not descendants of any 
of the nodes z\ , . . . , z n . 
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The idea, of course, is that the z\ describe the roots of the trees that contain the existentially 
quantified nodes xi, . . . ,x m in the original formula <p of S2(<). The straightforward proof 
of the claim is omitted. The finite disjunction ranges over all possible repartitions of the 
nodes x\, . . . , x m into distinct trees. □ 

From this normal form, since © distributes across union, it suffices to give a £2(<) 
formula for languages of the form 

L © Mi © • • • © M n © L'q © M[ © • • • © M' n , 

where the M languages are tree languages definable in Ti2(<) and where Lq,L' are forest 
languages definable in ITi(<). By the technique shown at the beginning of the proof, it is 
sufficient to obtain a formula for Lq © L' Q . But the language Lq © L' is closed under pieces, 
and therefore it is definable in IIi(<). □ 

Lemma 8.5. If context languages K,K' are definable in £2(<), then so is the commutative 
closure of K ■ K 1 . 

Proof. We write K K' for the commutative closure of K ■ K' . 

Consider first the case when either K or K' is a language {oO}. The formula places a 
variable on node corresponding to oD, and relativizes the formula for the remaining context 
language to the remaining nodes. 

Consider now the general case. Again, we use a normal form lemma for languages 



definable in S2(<). This lemma is prove the same way as Lemma 8.4 



Lemma 8.6. Every context language K definable in S2(<) can be written as a finite union 
of languages of the kinds 

K {an} (□ © L) or □ © L 
where K and L are context language and forest languages definable in E2(<). 



We now use Lemma 8.6 to finish the proof of Lemma 8.6 We want to show that KQK' 
is definable in Ea(<). We apply Lemma 8.6 to the languages K and K' . Since the operation 
distributes across union, we can assume that the unions describing K and K' use just 
one language, of either of the two kinds. We have four cases to consider, we only do the 
most difficult one 

K {aH} (□ © L) (□ © L') {£»'□} k' . 

This language is the same as 

K {aH} (□ © L © L') {a'D} K' . 

Let ipg be the Ti2(<) formula defining the context language K {aD}, obtained from 
the first case considered in the proof. Let ip^, be the Y<2(<) formula defining {a'D} K 1 
obtained in the same way. Both of these formulas have a free variable, which describes the 
hole of the context. Using Lemma 8.3 we also have a £2(<) formula ipL®L' defining L@L' . 

The desired S2(<) formula puts an existentially quantified variable x on the node 
corresponding to oO, another existentially quantified variable x' on the node corresponding 
to a'D, and then runs three subformulas, for K, □ © L © L', and K' , on the remaining 
nodes, appropriately relativizing the quantification. More specifically, this is the formula 

3x 3x a[x) A a (x) A parent(x, x) A <ffc{x) A ip^,(x') A (Pl®L'( x > x ') 



TREE LANGUAGES DEFINED IN FIRST-ORDER LOGIC WITH ONE QUANTIFIER ALTERNATION 25 



where parent(x,x') is the LTi formula stating that x is a proper ancestor of x' and there 
are no nodes in between, (pg(x) is constructed from ipg by relativizing all quantification 
to the ancestors of x, ipf>,(x) is constructed from if) a, by relativizing all quantification 
to the descendants of x, and (Pl®l'(x,x') is constructed from tpL®L' by relativizing all 
quantification to the nodes that are neither ancestor of x nor descendant of a;'. □ 

9. Discussion 

In this paper we considered a signature with the descendant and lexicographic orders. 
It would be interesting to know what happens in the presence of other predicates such as 
the closest common ancestor, next sibling or child. 

Probably the most natural continuation of this work would be an effective character- 
ization of 5^2(<) or £2(<,<iex)- Note that this would strengthen our result: a language 
L is definable in A2 if and only if both L and its complement are definable in £2- We 
conjecture that, as in the case for words [I], the characterization of £2(<,<i ex ) requires 
replacing the equivalence in the A2 identity by a one-sided implication, which says that a 
language definable in £2(<> <iex) is closed under replacing v u by v^wv^ , for w ■< v. 
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